test(ci): batched rpm and deb delivery through a single modify task [do not merge]#6306
Draft
kduret wants to merge 44 commits into
Draft
test(ci): batched rpm and deb delivery through a single modify task [do not merge]#6306kduret wants to merge 44 commits into
kduret wants to merge 44 commits into
Conversation
TEMP test branch (do not merge). In plugins.yml: - disable the Artifactory deliver-packages job (if: false) - force the Pulp deliver to stability: unstable, drop the stability gates (a PR runs with canary stability) and decouple the twin from test-plugins - add a marker file in the os::linux::snmp plugin packaging so get-plugins packages something to deliver All changes tagged TEMP(test-pulp-unstable). (cherry picked from commit 9174df1)
TEMP(test-pulp-unstable): reproduce the volume of the failed develop run (~30 packages per distrib) so the batched task wait is validated under a realistic load.
TEMP(test-pulp-unstable): unit-tests and test-plugins bring nothing to the batched-delivery validation and would consume up to 25 runners for the 31 marked plugins; fatpacker/package tolerate the skipped dependency.
TEMP(test-pulp-unstable): replace the 31 packaging markers with a touch of the common packaging template (changes_common=true), so get-plugins lists all 675 plugins and the batched Pulp delivery is exercised at full nightly scale (~675 packages per distrib across the 8 parallel deliver jobs).
… into MON-200796-plugins-test-batch-wait
TEMP(test-pulp-unstable) experiment: upload the rpm packages as unassociated content (repository-less create tasks parallelize across the pulp workers instead of serializing on the repository lock), resolve the created content hrefs (sha256 fallback for job re-runs), then add the whole batch to the repository with a single modify task. Two repository-serialized tasks per delivery (modify + publication) instead of one per package. Requires the reconciled rpm/packages access policy (delivery-tooling#209).
… into MON-200796-plugins-test-batch-modify
TEMP(test-pulp-unstable): same scope as the sibling batch-wait test for a direct comparison (675 packages per distrib).
…e [skip ci] The previous restriction hit the first matching matrix, which is the disabled artifactory job's; anchor on the twin's job name this time. [skip ci]: no point running until delivery-tooling#209 is merged (the orphan upload is rejected by the default access policy).
This reverts commit cf52ee2.
The retry window only covers publication propagation: once a round reads the published metadata and resolves nothing new while some packages already resolved, the remaining ones are not part of the publication at all (e.g. evicted by the retention policy) - report right away instead of burning the whole 5-minute window, and log the per-round progress.
…he shared deb repository
kduret
changed the title
…s from the shared deb repository" This reverts commit 252e87b.
Faster to decompress at install time than xz, supported by rpm >= 4.14 (el8 onwards).
One flaky HEAD out of hundreds (content-app/S3 hiccup) must not fail the whole verification.
The testing repositories keep every delivered version (and the shared deb repository holds every suite), so the module listing exceeds a single page; the fail-loud truncation guard becomes an actual pagination.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Experiment: batched delivery through a single
modifytask, rpm and deb, on top of the full-scale test setup (675 plugins, Pulp unstable only, Artifactory and test jobs disabled). Validated at scale: delivery phase ~5 min (rpm) and ~8 min (deb) for 675 packages, against 43 and 76 min with per-package repository tasks.Method
rpm (deliver-rpm.sh, TEMP)
modifytask, then publish: 2 repository-serialized tasks per delivery instead of ~675.deb (deliver-deb.sh, TEMP) — a repository-less deb upload ignores
distribution/component, and the release_components api is not listable by the ci user, so:modifyadding packages + associations, then a structured publication.Hardening picked up along the way (ported to the 4 feature branches)
create_publication: publication started in--backgroundand awaited with re-authenticating polling (pulp-cli's built-in wait dies on the ~5 min OIDC token when publishing a large repository).One-off maintenance carried on this branch (TEMP)
apt-pluginsrepository (14 042 → 890 packages), develop deliveries preserved.Prerequisites (all merged)
delivery-tooling#209, #213 (rpm policy), #214 (deb policies), #216/#217 (bounded repository versions + daily orphan cleanup), #219 (retention set at creation).
Comparison target
The sibling test PR #6305 measured the same full-scale delivery with per-package repository tasks (batched waits). Refs: MON-200796